Skip to content

[20_0_X] Move stored types in io_v1 namespace, reset class versions to 3#50968

Open
makortel wants to merge 2 commits into
cms-sw:masterfrom
makortel:evoIntegrationDataFormats
Open

[20_0_X] Move stored types in io_v1 namespace, reset class versions to 3#50968
makortel wants to merge 2 commits into
cms-sw:masterfrom
makortel:evoIntegrationDataFormats

Conversation

@makortel
Copy link
Copy Markdown
Contributor

PR description:

This PR integrates the data format changes prototyped and tested in the EVOLUTION_X branch to the master branch for 20_0_X. Following the strategy agreed in ORP 2026-05-05, it is opened already now to allow it to be reviewed such that it could be merged quickly when the master branch moves to 20_0_X.

This PR includes

  • move data types stored in files in workflow 34434.0 (plus those in PREMIX data tier) in io_v1 namespace to allow schema evolution for complex cases that ROOT's schema evolution can not support (e.g. pointed-to container type of edm::Ref being changed)
  • reset all class versions to 3 and all input-related read rules are removed (except of data types stored in RAW)
  • two earlier data type evolution attempt that were rejected at the time because they would have broken backwards compatibility

For more information please see the ORP 2026-05-05 presentation and the earlier talks linked there.

This PR is unusual in several ways:

Resolves cms-sw/framework-team#2227

PR validation:

None beyond the tests in the EVOLUTION_X IBs. Checked on CMSSW_17_0_X_2026-05-18-1100 that cherry-picking #50953, #50954, and this PR shows no difference wrt. CMSSW_17_0_EVOLUTION_X branch.

@makortel
Copy link
Copy Markdown
Contributor Author

hold

Until 20_0_X opens

@cmsbuild
Copy link
Copy Markdown
Contributor

cmsbuild commented May 18, 2026

cms-bot internal usage

@cmsbuild
Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50968/49372

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request has been put on hold by @makortel
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

@cmsbuild
Copy link
Copy Markdown
Contributor

A new Pull Request was created by @makortel for master.

It involves the following packages:

  • Alignment/CommonAlignmentProducer (alca)
  • Alignment/MillePedeAlignmentAlgorithm (alca)
  • AnalysisDataFormats/SUSYBSMObjects (analysis)
  • AnalysisDataFormats/TopObjects (analysis)
  • AnalysisDataFormats/TrackInfo (analysis)
  • DPGAnalysis/MuonTools (xpog)
  • DQM/DTMonitorModule (dqm)
  • DQM/GEM (dqm)
  • DQM/SiStripCommissioningSources (dqm)
  • DataFormats/Alignment (alca)
  • DataFormats/BTauReco (reconstruction)
  • DataFormats/BeamSpot (reconstruction)
  • DataFormats/CLHEP (core)
  • DataFormats/CSCDigi (simulation)
  • DataFormats/CSCRecHit (reconstruction)
  • DataFormats/CTPPSDigi (reconstruction, simulation)
  • DataFormats/CTPPSReco (reconstruction)
  • DataFormats/CaloRecHit (reconstruction)
  • DataFormats/CaloTowers (reconstruction)
  • DataFormats/Candidate (reconstruction)
  • DataFormats/CastorReco (reconstruction)
  • DataFormats/Common (core)
  • DataFormats/DTDigi (simulation)
  • DataFormats/DTRecHit (reconstruction)
  • DataFormats/EcalDetId (simulation)
  • DataFormats/EcalDigi (simulation)
  • DataFormats/EcalRawData (simulation)
  • DataFormats/EcalRecHit (reconstruction)
  • DataFormats/EgammaCandidates (reconstruction)
  • DataFormats/EgammaReco (reconstruction)
  • DataFormats/FEDRawData (daq)
  • DataFormats/FTLDigi (simulation)
  • DataFormats/FTLRecHit (reconstruction)
  • DataFormats/FWLite (core)
  • DataFormats/ForwardDetId (simulation)
  • DataFormats/GEMDigi (simulation)
  • DataFormats/GEMRecHit (reconstruction)
  • DataFormats/GeometryCommonDetAlgo (simulation)
  • DataFormats/GeometrySurface (simulation)
  • DataFormats/GeometryVector (simulation)
  • DataFormats/GsfTrackReco (reconstruction)
  • DataFormats/HGCDigi (simulation)
  • DataFormats/HGCRecHit (reconstruction)
  • DataFormats/HGCalDigi (simulation)
  • DataFormats/HGCalReco (reconstruction)
  • DataFormats/HLTReco (hlt)
  • DataFormats/HcalCalibObjects (alca)
  • DataFormats/HcalDetId (simulation)
  • DataFormats/HcalDigi (simulation)
  • DataFormats/HcalIsolatedTrack (reconstruction)
  • DataFormats/HcalRecHit (reconstruction)
  • DataFormats/HeavyIonEvent (reconstruction)
  • DataFormats/HepMCCandidate (generators)
  • DataFormats/JetMatching (simulation)
  • DataFormats/JetReco (reconstruction)
  • DataFormats/L1CSCTrackFinder (l1)
  • DataFormats/L1CaloTrigger (l1)
  • DataFormats/L1DTTrackFinder (l1)
  • DataFormats/L1GlobalCaloTrigger (l1)
  • DataFormats/L1GlobalMuonTrigger (l1)
  • DataFormats/L1GlobalTrigger (l1)
  • DataFormats/L1TCalorimeter (l1)
  • DataFormats/L1TCalorimeterPhase2 (l1)
  • DataFormats/L1TCorrelator (l1)
  • DataFormats/L1TGlobal (l1)
  • DataFormats/L1THGCal (l1)
  • DataFormats/L1TMuon (l1)
  • DataFormats/L1TMuonPhase2 (l1)
  • DataFormats/L1TParticleFlow (l1)
  • DataFormats/L1TrackTrigger (l1)
  • DataFormats/L1Trigger (l1)
  • DataFormats/LTCDigi (l1)
  • DataFormats/Luminosity (reconstruction)
  • DataFormats/METReco (reconstruction)
  • DataFormats/MuonDetId (simulation)
  • DataFormats/MuonReco (reconstruction)
  • DataFormats/MuonSeed (reconstruction)
  • DataFormats/NanoAOD (xpog)
  • DataFormats/OnlineMetaData (daq, reconstruction)
  • DataFormats/ParticleFlowCandidate (reconstruction)
  • DataFormats/ParticleFlowReco (reconstruction)
  • DataFormats/PatCandidates (reconstruction, xpog)
  • DataFormats/Phase2TrackerCluster (reconstruction)
  • DataFormats/Phase2TrackerDigi (reconstruction)
  • DataFormats/ProtonReco (reconstruction)
  • DataFormats/RPCDigi (simulation)
  • DataFormats/RPCRecHit (reconstruction)
  • DataFormats/RecoCandidate (reconstruction)
  • DataFormats/Scalers (daq, l1, reconstruction)
  • DataFormats/SiPixelCluster (reconstruction)
  • DataFormats/SiPixelDetId (simulation)
  • DataFormats/SiPixelDigi (simulation)
  • DataFormats/SiPixelRawData (simulation)
  • DataFormats/SiStripCluster (reconstruction)
  • DataFormats/SiStripCommon (reconstruction)
  • DataFormats/SiStripDetId (simulation)
  • DataFormats/SiStripDigi (simulation)
  • DataFormats/Streamer (core)
  • DataFormats/TCDS (daq, reconstruction)
  • DataFormats/TauReco (reconstruction)
  • DataFormats/TestObjects (core)
  • DataFormats/TotemReco (reconstruction)
  • DataFormats/TrackCandidate (reconstruction)
  • DataFormats/TrackReco (reconstruction)
  • DataFormats/TrackerCommon (reconstruction)
  • DataFormats/TrackerRecHit2D (reconstruction)
  • DataFormats/TrackingRecHit (reconstruction)
  • DataFormats/TrajectorySeed (reconstruction)
  • DataFormats/TrajectoryState (reconstruction)
  • DataFormats/V0Candidate (reconstruction)
  • DataFormats/VertexReco (reconstruction)
  • FWCore/Utilities (core)
  • FastSimDataFormats/External (fastsim)
  • FastSimDataFormats/NuclearInteractions (fastsim)
  • FastSimDataFormats/PileUpEvents (fastsim)
  • Fireworks/Calo (visualization)
  • Fireworks/Tracks (visualization)
  • HLTrigger/btau (hlt)
  • JetMETCorrections/Type1MET (reconstruction)
  • L1Trigger/GlobalTrigger (l1)
  • L1Trigger/GlobalTriggerAnalyzer (l1)
  • L1Trigger/L1THGCalUtilities (l1)
  • L1Trigger/L1TMuonBarrel (l1)
  • L1Trigger/Phase2L1ParticleFlow (l1)
  • PhysicsTools/NanoAOD (xpog)
  • PhysicsTools/SelectorUtils (reconstruction)
  • RecoEgamma/EgammaElectronProducers (reconstruction)
  • RecoLocalMuon/GEMCSCSegment (reconstruction)
  • RecoLocalMuon/RPCRecHit (reconstruction)
  • RecoMuon/MuonIdentification (reconstruction)
  • RecoMuon/MuonIsolation (reconstruction)
  • RecoMuon/TrackerSeedGenerator (reconstruction)
  • RecoTauTag/RecoTau (reconstruction)
  • RecoTracker/FinalTrackSelectors (reconstruction)
  • RecoTracker/TkSeedGenerator (reconstruction)
  • RecoTracker/TrackProducer (reconstruction)
  • SimDataFormats/Associations (simulation)
  • SimDataFormats/CaloAnalysis (simulation)
  • SimDataFormats/CaloHit (simulation)
  • SimDataFormats/CaloTest (simulation)
  • SimDataFormats/CrossingFrame (simulation)
  • SimDataFormats/DigiSimLinks (simulation)
  • SimDataFormats/EcalTestBeam (simulation)
  • SimDataFormats/EncodedEventId (simulation)
  • SimDataFormats/Forward (simulation)
  • SimDataFormats/GEMDigiSimLink (simulation)
  • SimDataFormats/GeneratorProducts (generators)
  • SimDataFormats/HcalTestBeam (simulation)
  • SimDataFormats/HiGenData (simulation)
  • SimDataFormats/PileupSummaryInfo (simulation)
  • SimDataFormats/RPCDigiSimLink (simulation)
  • SimDataFormats/RandomEngine (simulation)
  • SimDataFormats/Track (simulation)
  • SimDataFormats/TrackerDigiSimLink (simulation)
  • SimDataFormats/TrackingAnalysis (simulation)
  • SimDataFormats/TrackingHit (simulation)
  • SimDataFormats/ValidationFormats (simulation)
  • SimDataFormats/Vertex (simulation)
  • TBDataFormats/EcalTBObjects (analysis)
  • TBDataFormats/HcalTBObjects (analysis)
  • TauAnalysis/MCEmbeddingTools (simulation)
  • TrackingTools/GsfTracking (reconstruction)
  • Validation/RecoMuon (dqm)

@Alejandro1400, @BenjaminRS, @Dr15Jones, @JanChyczynski, @Martin-Grunewald, @Moanwar, @alja, @arunhep, @atpathak, @battibass, @civanch, @cmsbuild, @ctarricone, @emeschi, @ftorrresd, @gabrielmscampos, @jfernan2, @kpedro88, @lviliani, @makortel, @mandrenguyen, @mdhildreth, @mkirsano, @mmusich, @perrotta, @quinnanm, @rseidita, @sensrcn, @smorovic, @smuzaffar, @srimanob, @ssekmen, @theofil, @tvami can you please review it and eventually sign? Thanks.
@24LopezR, @AlexDeMoor, @CeliaFernandez, @Fedespring, @GiacomoSguazzoni, @HuguesBrun, @IzaakWN, @Martin-Grunewald, @Ming-Yan, @Prasant1993, @ReyerBand, @Sam-Harper, @Senphy, @VinInn, @VourMa, @a-kapoor, @abbiendi, @abdoulline, @afiqaize, @ahinzmann, @alesaggio, @alja, @amagitte, @amarini, @andrzejnovak, @apsallid, @argiro, @azotz, @battibass, @bellan, @bsunanda, @calderona, @castaned, @cericeci, @clelange, @cseez, @denizsun, @dgulhan, @dkotlins, @echabert, @elusian, @erikbutz, @eyigitba, @fabferro, @fabiocos, @felicepantaleo, @ferencek, @fsimone91, @giovanni-mocellin, @gkasieczka, @gouskos, @gpetruc, @grzanka, @hatakeyamak, @jainshilpi, @jbsauvan, @jdamgov, @jdolen, @jhgoh, @jlidrych, @jshlee, @lgray, @llunerti, @mandrenguyen, @mariadalfonso, @martinamalberti, @mbluj, @missirol, @mmarionncern, @mmasciov, @mmusich, @mroguljic, @mtosi, @nhanvtran, @pakhotin, @pavlo-kashko, @pfs, @philippgadow, @ptcox, @ram1123, @rappoccio, @rchatter, @robervalwalsh, @rociovilar, @rovere, @rsreds, @salimcerci, @sameasy, @sbein, @schoef, @seemasharmafnal, @sethzenz, @sobhatta, @sviret, @thomreis, @threus, @tlampen, @tocheng, @trocino, @tsusa, @valsdav, @vandreev11, @varuns23, @wang0jin, @watson-ij, @wddgit, @yetkinyilmaz, @yslai, @yuanchao this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@makortel
Copy link
Copy Markdown
Contributor Author

@cmsbuild
Copy link
Copy Markdown
Contributor

-1

Failed Tests: Build ClangBuild
Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3887be/53399/summary.html
COMMIT: 88e2876
CMSSW: CMSSW_17_0_X_2026-05-20-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50968/53399/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3887be/53399/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3887be/53399/git-merge-result

Failed Build

I found compilation error when building:

++ SCRAM_NOEDM_CHECKS=yes
++ /usr/bin/time -v scram build -v -k -j 16
***WARNING: Multiple usage of "DataFormatsL1DTTrackFinder". Please cleanup "lib" in "export" section of "src/DataFormats/L1DTTrackFinder/BuildFile.xml"
>> Local Products Rules ..... started
>> Local Products Rules ..... done
gmake: Circular tmp/el8_amd64_gcc13/cache/cuda_dlink/RecoTrackerLSTCoreCudaAsync <- tmp/el8_amd64_gcc13/cache/cuda_dlink/RecoTrackerLSTCoreCudaAsync dependency dropped.
gmake: Circular tmp/el8_amd64_gcc13/cache/rocm_dlink/RecoTrackerLSTCoreROCmAsync <- tmp/el8_amd64_gcc13/cache/rocm_dlink/RecoTrackerLSTCoreROCmAsync dependency dropped.
>> Building CMSSW version CMSSW_17_0_X_2026-05-20-1100 ----
>> Creating project symlinks
>> Entering Package Alignment/CommonAlignmentProducer
>> Entering Package CondFormats/Alignment


Failed Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package TopQuarkAnalysis/TopSkimming
>> Entering Package TopQuarkAnalysis/TopTools
>> Entering Package TrackPropagation/Geant4e
>> Entering Package Utilities/ReleaseScripts
>> Compile sequence completed for CMSSW CMSSW_17_0_X_2026-05-20-1100
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
Command exited with non-zero status 1
	Command being timed: "scram build -k -j 32 COMPILER=llvm compile BUILD_LOG=yes"
	User time (seconds): 10984.20
	System time (seconds): 1170.33
	Percent of CPU this job got: 1360%


The io_v1 namespace allows schema evolution of complex cases (e.g.
pointed-to container type of edm::Ref being changed) that ROOT's
schema evolution can not support.

Moving classes to io_v1 breaks backwards compatibility for these types
that was agreed to be done in CMSSW_20_0_0 that serves a start of
Phase 2 (only) targeting software.

Taking advantage of the backwards compatiblity break, the class
versions are reset to 3.

In addition, include two complex schema evolution cases that were too
difficult earlier
- Make PFCandidate to no longer inherit from CompositeCandidate
- Reduce use of RangeMap in muon code
@makortel
Copy link
Copy Markdown
Contributor Author

This change was needed because #49932 was merged recently and that added a forward declaration instead of #include "DataFormats/CaloRecHit/interface/CaloClusterFwd.h".

@cmsbuild
Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50968/49434

@cmsbuild
Copy link
Copy Markdown
Contributor

@makortel
Copy link
Copy Markdown
Contributor Author

@cmsbuild, please test

@cmsbuild
Copy link
Copy Markdown
Contributor

-1

Failed Tests: ClangBuild
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3887be/53414/summary.html
COMMIT: 7095d7a
CMSSW: CMSSW_17_0_X_2026-05-21-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50968/53414/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package TopQuarkAnalysis/TopSkimming
>> Entering Package TopQuarkAnalysis/TopTools
>> Entering Package TrackPropagation/Geant4e
>> Entering Package Utilities/ReleaseScripts
>> Compile sequence completed for CMSSW CMSSW_17_0_X_2026-05-21-1100
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
Command exited with non-zero status 1
	Command being timed: "scram build -k -j 32 COMPILER=llvm compile BUILD_LOG=yes"
	User time (seconds): 8229.75
	System time (seconds): 859.53
	Percent of CPU this job got: 1086%


@cmsbuild
Copy link
Copy Markdown
Contributor

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50968/49446

ERROR: Build errors found during clang-tidy run.

   12 |   void CaloCluster::setClusterFlag(ClusterFlag flag, bool val) {
      |        ~~~~~~~~~~~~~^
src/DataFormats/L1TCalorimeter/src/CaloCluster.cc:13:9: error: use of undeclared identifier 'val' [clang-diagnostic-error]
   13 |     if (val) {
      |         ^~~
src/DataFormats/L1TCalorimeter/src/CaloCluster.cc:14:7: error: use of undeclared identifier 'm_clusterFlags' [clang-diagnostic-error]
   14 |       m_clusterFlags |= (0x1 << flag);
      |       ^~~~~~~~~~~~~~
src/DataFormats/L1TCalorimeter/src/CaloCluster.cc:14:33: error: use of undeclared identifier 'flag' [clang-diagnostic-error]
   14 |       m_clusterFlags |= (0x1 << flag);
      |                                 ^~~~
src/DataFormats/L1TCalorimeter/src/CaloCluster.cc:16:7: error: use of undeclared identifier 'm_clusterFlags' [clang-diagnostic-error]
   16 |       m_clusterFlags &= ~(0x1 << flag);
      |       ^~~~~~~~~~~~~~
src/DataFormats/L1TCalorimeter/src/CaloCluster.cc:16:34: error: use of undeclared identifier 'flag' [clang-diagnostic-error]
   16 |       m_clusterFlags &= ~(0x1 << flag);
      |                                  ^~~~
src/DataFormats/L1TCalorimeter/src/CaloCluster.cc:20:8: error: use of undeclared identifier 'CaloCluster'; did you mean 'l1t::CaloCluster'? [clang-diagnostic-error]
   20 |   void CaloCluster::setHwPtEm(int pt) { m_hwPtEm = pt; }
      |        ^~~~~~~~~~~
      |        l1t::CaloCluster
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment